Image analysis system for identifying lung features

ABSTRACT

Methods and apparatuses for identifying lung features are provided herein.

INCORPORATION BY REFERENCE

An Application Data Sheet is filed concurrently with this specificationas part of the present application. Each application that the presentapplication claims benefit of or priority to as identified in theconcurrently filed Application Data Sheet is incorporated by referenceherein in its entirety and for all purposes.

FIELD

The embodiments of the present disclosure relate to the field of digitalimaging, and more particularly to processing a 3D digital medical imageof a lung to identify the boundaries of a plurality of lung segments.

BACKGROUND

Review and interpretation of 3D digital medical images, such as theimages produced by computed tomography (CT) scanners and magneticresonance imaging (MRI) machines, is a common task of surgeons and otherphysicians. Informed by their interpretation of a patient's medicalimage, a physician will make diagnoses and treatment decisions aboutthat patient. For a patient with a suspected cancerous lesion in theirlung, some of the questions considered by a physician will includewhether the lesion should be surgically removed, whether the lesionshould be ablated (which is to say, frozen or heated in order to destroythe cancerous tissue), or whether the patient should be treated byradiation therapy and/or chemotherapy.

For patients considered for surgery, a surgeon will evaluate thepatient's anatomy within the medical image to help decide what parts ofthe lung should be removed along with the cancer. The goals of thesurgeon are twofold: (1) to remove the suspected area of cancer in itsentirety, while (2) retaining as much healthy lung tissue as possible.In order to help ensure that cancer cells are not left behind, thesurgeon will remove an extra area of healthy tissue around the suspectedmalignancy, commonly referred to as a “margin.” The size of the marginconsidered oncologically appropriate varies according to characteristicsof the tumor, tissue, and organ type.

In deciding what type of operation to perform and how best to performit, the surgeon will rely largely on the medical imaging of the patient(e.g., CT scan or MRI). Increasingly, surgeons are also utilizingthree-dimensional (3D) reconstructions of such medical images. The useof 3D reconstructions during surgical planning has been shown to providesurgeons with a better understanding of the patient's anatomy and toresult in improved surgical outcomes. Examples of such benefits aredescribed by Shirk 2019. Presently, 3D reconstructions of lungstypically include a depiction of the lung surface, the lung lobes, andother anatomical structures such as lesions, airways and blood vessels.Existing 3D reconstruction techniques do not, however, include anautomatically generated depiction of the segments within the lobes.

SUMMARY

Briefly, in certain embodiments, a method for identifying segments in alung including a number of lobes, each lobe including a number of thesegments, each segment having a boundary, may include receiving imagedata forming a three-dimensional representation of at least a part ofthe lung, computationally identifying, using the image data, (i) lungparenchyma and/or an outer surface of the lung and (ii) at least oneanatomical structure within the lung, where the at least one anatomicalstructure is one or more fissures between the lobes, one or more veins,one or more arteries, and/or one or more airways, computationallyidentifying, from (i) the lung parenchyma and/or the outer surface ofthe lung and (ii) the at least one identified anatomical structure,substantially all the boundary of at least one segment within the lung,where computationally identifying substantially all of the boundary ofthe at least one segment within the lung includes computationallyidentifying, from the at least one identified anatomical structure,substantially all of a segment-to-segment surface boundary between theat least one segment and an additional segment within the lung, theadditional segment being adjacent to the at least one segment and thesegment-to-segment surface boundary lying on the outer surface of thelung, and generating a representation containing substantially all theboundary of the at least one segment within the lung.

In some embodiments, the image data includes a CT scan and/or an Mill.In some embodiments, the CT scan and/or MM include a CT scan withcontrast and/or an MM with contrast. In some embodiments, the methodalso includes providing a visual presentation containing substantiallyall the boundary of the at least one segment within the lung. In someembodiments, the at least one anatomical structure includes the one ormore veins. In some embodiments, the at least one anatomical structureincludes the one or more arteries. In some embodiments, the at least oneanatomical structure includes the one or more airways. In someembodiments, computationally identifying (i) lung parenchyma and/or anouter surface of the lung includes computationally identifying one ormore fissures between lobes and the at least one anatomical structureincludes the one or more veins. In some embodiments, computationallyidentifying (i) lung parenchyma and/or an outer surface of the lungincluding computationally identifying one or more fissures between lobesand the at least one anatomical structure includes the one or morearteries. In some embodiments, computationally identifying (i) lungparenchyma and/or an outer surface of the lung includes computationallyidentifying one or more fissures between lobes and the at least oneanatomical structure includes the one or more airways. In someembodiments, the at least one anatomical structure includes the one ormore veins and the one or more arteries. In some embodiments, the atleast one anatomical structure includes the one or more veins and one ormore airways. In some embodiments, the at least one anatomical structureincludes the one or more arteries and the one or more airways. In someincludes, the at least one segment includes a given segment, generatingthe representation includes determining a location of a first anatomicalstructure within the given segment, and the first anatomical structureis at least a portion of the one or more veins, at least a portion ofthe one or more arteries, and/or at least a portion of the one or moreairways. In some embodiments, the at least one segment includes a givensegment, generating the representation includes determining a locationof a first anatomical structure within the given segment, and the firstanatomical structure is at least a portion of one or more intersegmentalveins. In some embodiments, the at least one segment includes a givensegment and generating the representation includes determining alocation of at least one or more lesions within the given segment. Insome embodiments, the at least one segment includes a given segment andgenerating the representation includes determining a location of atleast one or more lymph nodes within the given segment. In someembodiments, the at least one segment includes a given segment and themethod may further include computationally determining, using the imagedata, a placement of a first anatomical structure within the boundary ofthe given segment, where the first anatomical structure is at least aportion of the one or more veins, at least a portion of the one or morearteries, at least a portion of the one or more airways, one or morelesions, and/or one or more lymph nodes. In some embodiments, the methodfurther includes computationally identifying, using the image data, oneor more intersegmental veins within the lung, where computationallyidentifying substantially all the boundary of at least one segmentwithin the lung includes computationally identifying substantially allthe boundary of at least one segment based at least in part on theidentified one or more intersegmental veins. In some embodiments, thelung has an outer surface and the method further includescomputationally identifying, using the image data, at least portions ofthe outer surface of the lung, where computationally identifyingsubstantially all the boundary of at least one segment within the lungincludes computationally identifying substantially all the boundary ofat least one segment based at least in part on the identified portionsof the outer surface of the lung. In some embodiments, the lung furtherincludes parenchyma and the method further includes computationallyidentifying, using the image data, at least portions of the parenchyma,where computationally identifying substantially all the boundary of atleast one segment within the lung includes computationally identifyingsubstantially all the boundary of at least one segment based at least inpart on the identified portions of the parenchyma. In some embodiments,the at least one segment includes a given segment and the method alsoincludes, calculating, based on the computationally identified boundaryof the given segment, a volume of the given segment. In someembodiments, the at least one segment includes a given segment and themethod further includes calculating a distance between any boundary ofthe given segment and another anatomical structure within the lung. Insome embodiments, the method further includes calculating adistance-based measure of the at least one segment, where thedistance-based measure is a maximum diameter, a centroid, a boundingbox, a surface area of the boundaries of the at least one segment, alength of the at least one segment on a given surface of the lung, amaximum length of the at least one segment on the surface of the lung,and/or a surface area or length of where the at least one segment meetsa second segment. In some embodiments, the at least one segment includesa given segment, the lung has an outer surface, and the method furtherincludes calculating a minimum distance between the given segment andthe outer surface of the lung. In some embodiments, the image dataincludes image data obtained with at least one of a CT scan, an MRI, anultrasound scan, and a nuclear medicine scan.

In certain embodiments, a method for identifying features in a lungincluding a number of lobes, each lobe including a number of segments,each segment having a boundary, may include receiving image data forminga volumetric representation of at least a part of a human lung,computationally identifying, using the image data, an anatomical featurewithin the lung, where the anatomical feature is one or more fissuresbetween the lobes, a network of veins, a network of arteries, a networkof airways, and/or one or more intersegmental veins, computationallyidentifying, using the anatomical feature, a boundary of at least onesegment within the lung, and generating a representation containing (i)substantially all the boundary of the at least one segment within thelung, and (ii) the one or more fissures between the lobes, the networkof veins, the network of arteries, the network of bronchi, the one ormore intersegmental veins, or any combination of the foregoing.

In certain embodiments, a method for identifying features in a humanlung including a number of lobes, each lobe including a number ofsegments, may include receiving image data forming a volumetricrepresentation of at least a part of the human lung, computationallyidentifying, using the image data, portions of at least two lobes and afissure between said two lobes; computationally identifying, using theimage data, a network of arteries, a network of veins, and/or a networkof bronchi, where identifying the network of arteries, veins, and/orbronchi includes computationally identifying a tube-like structure inimage data, where the tube-like structure is identified by identifying aset of gradient changes within the image data, and computationallydetermining, based on how the tube-like structure branches within thehuman lung, that the tube-like structure is part of the network ofarteries, veins, and/or bronchi, and computationally identifying, basedon the identified network of arteries, veins, and/or bronchi and theidentified lobes or the fissure between said lobes, boundaries of aplurality of segments within at least one of said lobes.

In some embodiments computationally identifying the boundary of a singlesegment in the plurality of segments includes computationallyidentifying a volume within image data that exclusively receives bloodfrom a single branch of the network of arteries. In some embodiments,computationally identifying the boundary of a single segment in theplurality of segments includes computationally identifying a volumewithin the image data that does not pass through the fissure betweensaid lobes. In some embodiments, the method also includescomputationally identifying, using the image data, a lesion within thehuman lung, computationally determining, from the image data, that thelesion is located in a given segment of the plurality of segments, andcomputationally measuring, from the image data, a minimum distancebetween the lesion and the boundary of the given segment.

In certain embodiments, a method for identifying features in a humanlung including a number of lobes, each lobe including a number ofsegments, may include receiving image data forming a volumetricrepresentation of at least a part of a human lung, computationallyidentifying, using the image data, a network of arteries, whereidentifying the network of arteries includes computationally identifyinga tube-like structure in the image data, where the tube-like structureis identified by identifying a set of gradient changes within the imagedata, and computationally determining, using the image data and based onhow the tube-like structure branches within the human lung, that thetube-like structure is part of the network of arteries, computationallyidentifying, using the image data, a network of bronchi, whereidentifying the network of bronchi includes computationally identifyingan additional tube-like structure in the image data, where theadditional tube-like structure is identified by identifying anadditional set of gradient changes within the image data, andcomputationally determining, using the image data and based on how theadditional tube-like structure branches within the human lung, that theadditional tube-like structure is part of the network of bronchi, andcomputationally identifying, based on the identified network of arteriesand the identified network of bronchi, boundaries of a plurality ofsegments within at least one of said lobes.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1-4 are images of a human lung.

FIGS. 5A and 5B are a process flow diagram depicting operations ofmethods performed in accordance with certain disclosed embodiments.

FIGS. 6A and 6B are a process flow diagram depicting operations ofmethods performed in accordance with certain disclosed embodiments.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth toprovide a thorough understanding of the presented embodiments. Thedisclosed embodiments may be practiced without some or all of thesespecific details. In other instances, well-known process operations havenot been described in detail to not unnecessarily obscure the disclosedembodiments. While the disclosed embodiments will be described inconjunction with the specific embodiments, it will be understood that itis not intended to limit the disclosed embodiments.

I. Lung Anatomy

The lung anatomy of a patient discernible within medical images includesthe lung's airways, pulmonary veins (blood vessels that carry oxygenatedblood), and pulmonary arteries (blood vessels which carry deoxygenatedblood), each of which form a tree-like system or network within eachlung. The veins and arteries can be more generally referred to asvessels or vasculature.

Each lung is composed of multiple lobes. Each lobe has its ownindependent and distinct subtree of each of the three branching systems(airways, vein, and artery). Pulmonary arteries, veins, and airways donot cross lobe boundaries. A branch upstream of all veins in a lobe issometimes referred to as a lobar branch of the pulmonary arteries. Abranch upstream all airways in a lobe is sometimes referred to as alobar branch of the lobar airway. The lobes are separated by fissureswhich are discernable on some medical images. Lung tissue which is not ablood vessel or an airway is called parenchyma.

In human anatomy, the left lung is smaller than the right, and isseparated into two lobes: an upper lobe and a lower lobe. The right lungconsists of three lobes: upper, lower, and middle.

An image of a right lung is shown in FIG. 1. As shown in FIG. 1, thereis a relatively large amount of space (illustrated as regions 110)between the arterial subtrees of the upper lobe 102 and the middle lobe104 and the arterial subtree of the lower lobe 106 (lobe parenchyma isnot shown in FIG. 1).

Each lobe can further be subdivided into segments. Similar to lobes,each segment also has an independent supply of blood and an independentbranch of the bronchus. Analogous to lobes, one segment within a lobecan be removed, leaving the other segments as functional units of thelung. Unlike lobes, however, segments are typically not separated byfissures. In rare cases, “accessory” fissures are visible in medicalimages and demarcate the border between two segments within a lobe.Aspects of this disclosure assume the common case, that no accessoryfissures are present within the medical image being processed.

In the left lung, each lobe contains four segments, for a total of eightsegments in the lung. In the right lung, there are a total of tensegments, with five in the lower lobe (four of which are referred to asthe “basal” segments”), two in the middle lobe, and three in the upperlobe.

An image of the upper lobe of a right lung is shown in FIG. 2. FIG. 2illustrates the spaces between the three segments of the upper lobe ofthe right lung. In FIG. 2, outline 202 shows the spaces around thearterial subtree of the apical segment, outline 204 shows the spacesaround the arterial subtree of the anterior segment, and outline 206shows the spaces around the arterial subtree of the posterior segment.

An image of a right lung is shown in FIG. 3. FIG. 3 illustrates the veinnetwork 302, the arterial network 304, the middle right lobe 306, thelower right lobe 308, and the three segments 310 a, 310 b, and 310 b ofthe upper right lobe.

An image of a right lung is shown in FIG. 4. FIG. 4 illustrates thearterial network 304, the right lower lobe, the right middle lobe, thethree segments (posterior, apical, and anterior) of the right upperlobe. While the networks of airways and veins are not explicitlyillustrated in FIG. 4, their absence is visible in FIG. 4 (e.g., thereare impressions of the veins and airways visible in FIG. 4).

Segments are far more difficult to discern on medical imaging thanlobes. Unlike lobes, segments are not separated by visible fissures.Without this visual cue, physicians must identify segments based ontheir understanding of vessels and airways that define segments.Specifically, one definition of the area of a segment is that it is thearea of a lobe which corresponds to a branch of the lobar pulmonaryartery (segmental artery) and a branch of the lobar airway. Each part ofthe lung parenchyma belongs to one segment, which means it is also fedby one branch of the airway and one branch of the pulmonary artery. Justas the subtrees of arteries and airways are distinct and independentbetween different lobes, within lobes the segmental subtrees of arteriesand airways are distinct and independent from the subtrees of arteriesand airways of other segments. Some veins may reside entirely within asegment, while other veins, sometimes referred to as intersegmentalveins pass along a portion of a boundary between lung segments. Thereare considerable variations in anatomy between human beings in thestructure and branching of segment-level vessels, which complicatessegment identification, as identification must consider the possiblevariations in vascular anatomy in human beings. Because of theforegoing, the identification and understanding of lung segmentsrequires a high level of training and expertise, and even the mostsophisticated physicians are commonly challenged with this exercise.

II. Lung Cancer Operations

The operation performed by the surgeon will have a different nameaccording to the nature and quantity of the lung which is removed.

A pneumonectomy is the removal of an entire lung.

A lobectomy is the removal of a lung lobe. A lobectomy leaves a patientwith more lung tissue than a pneumonectomy, as the lung will remainfunctional so long as at least one lobe remains. This is considered thestandard of care for surgery for most lung lesions larger than 2 cm indiameter.

A segmentectomy is the removal of one or more segments. A segmentectomyis sometimes referred to as an “anatomical resection” in that theresection lines follow segment boundaries of the patient anatomy.Segmentectomy is an approach often favored by experienced surgeons forsmaller lung lesions, typically 2 cm in diameter or smaller, whenpossible. The belief by many surgeons is that, for smaller lesions, asegmentectomy strikes the optimal balance between ensuring that allcancer is removed while preserving as much lung function as possible. Asegmentectomy requires the surgeon to confidently understand whichsegment is involved, and that an adequate margin will be obtained byremoving it. In some instances, multiple segments may be involvedbecause either the mass or the margin crosses over them. In these cases,the surgeon may remove both segments, and this would still be considereda segmentectomy. Also, some types of segmentectomies are more difficultthan others, often driven by how difficult it is for the surgeon tophysically access the particular segment.

A wedge resection is the removal of a triangle-shaped portion of thelung that is not defined by a lobe or segment. As opposed to asegmentectomy, a wedge resection is a “non-anatomical” resection, as theresection lines do not follow anatomical boundaries. Wedge is not thefavored approach, for oncological and/or functional reasons. Wedge istypically done in circumstances when a segmentectomy is not possible forsome reason.

III. Use of Medical Imaging in Surgical Planning

When deciding on surgical approach (e.g., lobectomy vs. segmentectomyvs. wedge), the surgeon will rely largely on the patient's medicalimage, e.g., a CT scan, an MM, and/or a PET scan. The surgeon willconsider, among other things, the lesion and location. In consideringlocation, the surgeon will identify the relevant lobe(s) and potentiallyalso the relevant segment(s). Identifying the relevant lobe is aprerequisite to a lobectomy and, as previously noted, is relatively easydue to the presence of visible fissures. Identifying the relevantsegment is a prerequisite to a segmentectomy and, as also previouslynoted, is relatively difficult due to the absence of visible fissuresseparating the segments.

In addition to looking at the patient's CT scans or other medicalimages, the surgeon may also look at a 3D reconstruction of such images.Presently, 3D reconstructions of lungs will typically include anautomatically-generated visual depiction of lobes, as defined byfissures, but lack an automatically-generated visual depiction ofsegments.

Although inclusion of lung segments within 3D reconstructions would beextremely helpful for surgeons, this is very difficult to achieve due tothe wide variability in human segmental anatomy and the way segments arecurrently identified by physicians, as more fully detailed below.

The present disclosure enables a computer system configured to identifyand label lung segments from a three-dimensional medical image such as aCT scan, and to visually depict such lung segments within a 3Dreconstruction. This visual depiction of lung segments will helpsurgeons decide whether a segmentectomy is a possible intervention forthe patient, and if so, which segment(s) should be removed, and the bestapproach to removing them. Certain details furthering that understandingmay include: (1) the knowledge of which segment(s) contain a lesion; (2)the distances between the lesion and the borders of the segment(s)considered for removal, to help evaluate prospective surgical margins;(3) the spatial relationship of the segments to each other, to aid inplanning and understanding the technical aspects of the surgery and toevaluate different possible surgical techniques.

IV. Use of Machine Learning in Medical Imaging

Various machine learning techniques are being employed towards medicalimaging. To date, such techniques have largely been applied towardsdiagnosis, in particular the detection and identification of aparticular disease state, such as a cancerous lesion. However, machinelearning techniques, such as supervised learning, may also be appliedtowards the process of image segmentation.

Supervised learning is useful where an output (such as a segmentation ofan image or a label of a segmentation) is available for a trainingdataset (e.g., a plurality of medical images). For example, a machinelearning model may be trained with various collections ofthree-dimensional image data taken from CT scans and/or MRIs, and inthese images, particular collections of voxels may be prelabeled asparenchyma, arteries, veins, fissures, airways, lesions, or lymph nodes.Using this training data, a machine learning model is trained to receiveunlabeled and optionally unsegmented image data as an input and providea segmented, labeled data set as an output. The output may identifycollections of voxels that are labeled as parenchyma, arteries, veins,lung surface, fissures, airways, lesions, lymph nodes, or anycombination thereof. In another approach, a machine learning model istrained with various collections of three-dimensional image datacontaining labelled collections of voxels (e.g., parenchyma, arteries,veins, lung surface, fissures, airways, lesions, and/or lymph nodes) andsegment boundaries for lung images. Using this training data, a machinelearning model is trained to receive labeled image data as inputs andproduce fully or substantially fully defined segment boundaries for thelungs.

Examples of machine learning algorithms that are supervised include butare not limited to linear regression, logistic regression, decisiontrees, support vector machine (SVM), naive Bayes, k-nearest neighbors,and neural networks (multilayer perceptron). Reinforcement learningfalls between supervised and unsupervised learning, where some feedbackis available for each predictive step or action but there is no preciselabel. Rather than being presented with correct input/output pairs as insupervised learning, a given input is mapped to a reward function thatan agent is trying to maximize. An example of a machine learningalgorithm that is reinforcement-based includes a Markov DecisionProcess. Other types of learning that may fall into the one or more ofthe categories described above include, for example, deep learning andartificial neural networks (e.g., convolutional neural networks).

Various training tools or frameworks may exist to train the machinelearning model. Examples of proprietary training tools include but arenot limited to Amazon Machine Learning, Microsoft Azure Machine LearningStudio, DistBelief, Microsoft Cognitive Toolkit. Examples of open sourcetraining tools include but are not limited to Apache Singa, Caffe, H2O,PyTorch, MLPACK, Google TensorFlow, Torch, and Accord.Net.

The trained machine learning model may take one of several forms. Insome implementations, the trained machine learning model is aclassification and regression tree or a random forest tree. In someimplementations, the trained machine learning model is an artificialneural network, such as a convolutional neural network. In someimplementations, the trained machine learning model is a linearclassifier, such as a linear regression, logistic regression, or supportvector machine.

V. Image Analysis System

This disclosure relates to systems and methods that generate, from oneor more three-dimensional (3D) digital images of a lung, arepresentation of the boundaries of the segments within that lung. Oncedefined, the boundaries can also be represented visually or used tocalculate volume measurements (e.g. the volume of a segment), distancemeasurements (e.g. the maximum diameter of the segment, distancesbetween a lesion or other structure and one or more segment boundaries),or other analyses based on the three-dimensional representation of thesegment's boundaries.

The types of medical images that may be used to generate the 3D digitalimage include any relevant medical image, including, but not limited to,a computed tomography scan (commonly referred to as a CT scan), amagnetic resonance imaging scan (commonly referred to as an MRI), anuclear medicine image including, but not limited to, a positronemission tomography scan (commonly referred to as a PET scan), and/or anultrasound image, and a three-dimensional reconstruction of any of theforegoing.

VI. Terminology

As used herein, the term “surgery” refers to a procedure for treatinginjuries, disorders, and other medical conditions by incision andassociated manipulation, particularly with surgical instruments. Theterms “operation,” “surgical operation,” and “surgery” are usedthroughout this disclosure. Unless otherwise clear from context, theterms are used interchangeably.

As used herein, an “image” refers to a visible depiction or otherrepresentation of an item such as the chest area of a patient. Invarious embodiments presented herein, images provide representations ofmorphologies and/or compositions of tissue, bone, or organs in asubject. Such images are sometimes referred to herein as medical images.Medical images of varying modality include without limitationrepresentations of tissue, bone, and/or organs derived from computerizedtomography (CT), magnetic resonance imaging (MM), x-ray imaging,positron emission tomography (PET) and other nuclear medicine scans,ultrasound imaging, and two-dimensional and/or three-dimensionalreconstructions of any of the foregoing. An image may be composed ofand/or transformed to three-dimensional image data formed of voxels.

As used herein, a “lung” refers to a single lung organ as present in anyvertebrate animal.

As used herein, a “pulmonary artery” is a blood vessel that carriesdeoxygenated blood into the lung.

As used herein, a “pulmonary vein” is a blood vessel that carriesoxygenated blood out of the lung.

As used herein, “vasculature” or “blood vessel” may be usedinterchangeably to refer to either a pulmonary artery or a pulmonaryvein.

As used herein, a lung “airway” refers to an anatomical structure thatcarries air into and out of the lung. Parts of the airway include thetrachea, the bronchi, and the bronchioles. Airways as used herein referboth to the “lumen”, or inside space of the airway, which is composed ofair, and the tissue wall of airway surrounding the lumen.

As used herein, a “lobe” refers to a lobe within a lung.

As used herein, a “segment” refers to a segment within a lobe.

As used herein, the lung “surface” refers to the exterior surface of alung.

As used herein, a lung “lesion” refers to any abnormal tissue within thelung, which may be malignant, benign, or of uncertain malignancy. Hereinthe term lesion may be used interchangeably with the term “mass,” whichis typically used in the context of larger lung lesions, and “nodule,”which is typically used in the context of a smaller lung lesion.

As used herein, the lung “parenchyma” refers to lung tissue which is notan airway, a pulmonary artery, or a pulmonary vein.

As used herein, the term “segmentation” refers to the process ofidentifying a structure of interest within a medical image, e.g., ananatomical structure such as an organ, lesion, or blood vessel. Whensegmentation occurs in a three-dimensional medical image, athree-dimensional representation of a structure of interest is created,which may be referred to as a “3D reconstruction”.

As used herein, the term “labeling” refers to the process of identifyingthe nature of a structure that has been segmented and assigning arelevant semantic name to such structure.

As used herein, the term “voxel,” or volumetric pixel, refers to asingle, discrete point in a regular grid in three-dimensional space. Inmedical images, voxels are typically rectangular prisms, and each has anintensity value (corresponding to a value on a grey or multi-coloredscale).

As used herein, the term “plane” refers to a surface in a Euclidean orother space. The surface has no volume. It need not be “flat” as with aEuclidean plane.

VII. Identification of Anatomical Structures within Medical Images(Excluding Lung Segments)

This section describes various examples of methods to segment and labelanatomical structures within medical images of lungs.

Among the relevant structures within medical images of lungs are: lungsurfaces, lung parenchyma, pulmonary arteries, pulmonary veins, airways,lung lobes, and lung fissures. The representations of some or all ofthese structures are inputs to the system and methods described inSection VIII which identify the boundaries between lung segments andwhich are the subject of the present disclosure.

The airways and pulmonary vasculature of the lung each form a tree-likebranching network of tubular structures. One method for segmentingairways and vasculature from three-dimensional medical images is basedon the mathematical properties of these tube-like structures. Suchmethod may, in some implementations, merely segment, but not label,tube-like anatomical features. Labeling may be accomplished in asubsequent operation. Methods that analyze images based on theseproperties include, without limitation, graph-based optimization,Hessian calculations, and flux-based methods. These techniques identifyregions of voxels (e.g. a ten by ten by ten cube of voxels within alarger image) where a mathematical operation on that region, such as aHessian calculation, returns a positive result indicating that the highintensity voxels within that region are in a tubular shape. Examples ofsuch techniques are described by Benmansour 2011, Benmansour 2013,Antiga 2007, Graham 2010, and Helmberger 2014.

Another method to segment airways and pulmonary vasculature is based ona machine learning model, which may also be used to merely segment orsegment and label such structures. An appropriate training algorithm,based on any one of several supervised machine learning algorithms, maybe used to train a machine learning model using information such asprior segmentations and labelings of airways and vessels. The trainingalgorithm may be used to recognize patterns in the images to accuratelyidentify airways and vessels and/or assign labels to such structures. Anexample of such technique is described by Ochs 2007.

In some implementations, structures that have been segmented as tubularstructures, such as by using graph-based optimization, Hessiancalculations, or flux-based methods, may be analyzed and assembled intoa tree-like structure that represents an interconnected system of veins,arteries, and/or airways. One approach to such analysis may involve theuse of a non-linear integer program to be solved by a non-linear solver.Another approach may utilize a machine learning model. In general, theseapproaches fit a number of candidate voxels to a model of a tree-likestructure. When a non-linear solver is used, such model is representedby non-linear equations; when machine learning is used, the model is themachine learning model which has been trained with examples of tree-likestructures. Examples of similar techniques are described by Turetken2013, Turetken 2016, Payer 2015, and Payer 2016.

Once a tree-like structure has been assembled, a separate technique suchas a machine learning model may be used to label the applicablestructures comprising such tree-like structure. For example, suchtechnique may receive partially processed image data containing one ormore pre-segmented tree-like anatomical structures that could be veins,arteries, and/or airways. The technique then analyzes the tree-likestructures and classifies or otherwise labels them as veins, arteries,and/or airways.

Aside from machine learning, another method to label a structures thatincludes an assembled tree-like structure is to take, as input, anon-linear integer program that represents the segmented artery, vein,and airway structures. That program is analyzed by a non-linear solverto assign labels to each of the segmented structures. An example of suchtechnique is described by Payer 2016.

As an alternative to labeling structures only after they have beensegmented, other methods that may be used to segment and labelstructures during the same process. For example, one technique which isspecialized for the segmentation of airways will generate segmentedstructures that are labeled as airways. Examples of such techniques aredescribed by van Ginneken 2008 and Meng 2018, Pu 2010.

In some embodiments, lung anatomical structures are identified usingatlas segmentation, which is a technique that can be used for thesegmentation and labeling not only for airways and vessels but also formany other anatomical features such as lung surface. For the remainderof this paragraph, segmentation may refer equally to segmentation onlyor to segmentation and labelling as part of the same process. In atlassegmentation, an image to be segmented (input) is compared against eachimage contained in a database of previously segmented images (theatlas). Atlas segmentation will identify the image within the atlaswhich is most “similar” to the input image. In this instance,“similarity” is based on the degree of transformation required to“register” the input image to the image in the atlas (meaning, align oneimage to the other). The smaller the degree of transformation requiredbetween two images, the more similar those two images are. Multiplesimilarity metrics are possible, including sum of squared distances,correlation coefficients, and mutual information. Once the most similaratlas image is identified, the nature of the transformation between theatlas image and the input image is determined. Multiple transformationmodels are possible, including rigid, affine, and deformabletransformation models. Once the nature of the transformation isdetermined, that transformation is applied to the segmentation of theatlas image which results in a segmentation of the input image. Anexample of such technique is described by Rohlfing 2004.

This paragraph and a few following paragraphs discuss optionaltechniques for the identification of the lung surface. One method toidentify the surface of the lung utilizes atlas segmentation. Itcompares the input image of the lung to an atlas of previously segmentedlung images, determining the most similar atlas lung image, thendetermining the transformation from the atlas image to the input image,then applying that transformation to the segmentation of the atlas imageto generate the segmentation of the input image lung surface.

Another method to identify the surface of the lung is through voxelintensity gradient analysis. This analysis considers voxel intensitygradients within the image, meaning, changes in voxel intensity over arange of contiguous voxels within an image. In CT scans and MM images,lung parenchyma shows up as voxels of low intensity, and the exterior ofthe lung surface shows up as voxels of higher intensity. A rapidlychanging gradient is indicative of a lung surface.

Another method to identify the surface of the lung is to utilize amachine learning model. This machine learning model would be trainedwith segmentations of the lung surface, along with the images from whichthose segmentations were generated. It would analyze the voxels withinan input image to accurately predict which voxels correspond to the lungsurface.

This paragraph and the next describe the identification of lungfissures. One method to identify fissures is through voxel intensityanalysis. Fissures can be identified based on gradient analysis, withthe analysis identifying voxel intensity changes between parenchymavoxels and fissure voxels, or by identifying voxels that constituteplane-like structures with hessian eigenvalue and eigenvector analysis.An example of a similar technique is described by Lassen 2013.

Another method to identify fissures involves the detection of “planes”(which are three-dimensional surfaces without a volume) throughvasculature and/or airway voxel density analysis. Fissures can berepresented as planes. One property of fissures is that the density ofvasculature and airways is lower around fissures than around parenchyma.One method of finding fissures in lungs is to determine which planeshave the lowest density of surrounding voxels which correspond toairways, arteries, and veins. Those planes may be taken to be therepresentation of the fissures, or may be adjusted in space to alignwith fissures identified using the gradient-based method described inthe preceding paragraph.

This paragraph and the next two discuss the identification of lungparenchyma. One method to identify the lung parenchyma is asubtraction-based method. This method starts by identifying the lungsurface and the three-dimensional space occupied within such lungsurface. From this three-dimensional space, the three-dimensional spaceoccupied by the vessels, airways and fissures is removed (subtracted).The resulting difference represents a segmentation of the lungparenchyma.

Another method to identify lung parenchyma is via analysis of voxelintensity variation and connectivity, which is the way in which voxelsrelate to adjacent voxels of similar intensity. This analysis involvesidentifying voxels that are low in intensity value and which are highlyconnected to other voxels also low in intensity value. Low intensityvoxels associated with airway lumens are then removed.

Another method to identify the lung parenchyma is through Otsu's methodof thresholding. Otsu's method determines a single intensity thresholdthat separates voxels into “foreground” and “background” voxels whileminimizing variance between the two groups; in the case of images oflungs, those correspond to low intensity parenchyma voxels within thesurface and high intensity voxels exterior to the lung surface. Anexample of such technique is described by Helen 2011.

The remaining paragraphs within this section discuss the identificationof lung lobes. One method to identify lung lobes is to start with (a)the segmented lung parenchyma and/or (b) the identified lung surface,and then subdividing the applicable structure according to identifiedlung fissures in either or both (a) and (b).

Another method to identify lung lobes employs an algorithm such as awatershed algorithm to pulmonary arteries. The method takes as input asegmentation of the pulmonary arteries. It then identifies each lobarbranch of the pulmonary arteries (and the subtree associated with suchbranch) using the following approach: arterial branches that appearwithin the segmented pulmonary arteries are matched to common branchingvariants of lung anatomy contained within predefined abstractrepresentations of arterial anatomy. Alternatively, lobar branches ofpulmonary arteries may be identified via a k-means clustering algorithm,where the branches of the tree are partitioned into multiple clusters,and each cluster corresponds to a different lobar branch. Areasidentified as artery subtrees are considered “high cost” areas within a“cost image” which is used as an input to a watershed algorithm todetermine the planes which are equidistant to the edges of each of thesubtrees, or, analogously, which are “lowest cost” by the input costimage. An example of such technique is described by Beucher 1992.

Another method to identify lung lobes is based on the application of awatershed algorithm to pulmonary veins. It takes as input a segmentationof the pulmonary veins. It then identifies each lobar branch of thepulmonary veins (and the subtree associated with such branch) using thefollowing approach: venous branches that appear within the segmentedpulmonary veins are matched to common branching variants of lung anatomycontained within predefined abstract representations of venous anatomy.Alternatively, lobar branches of pulmonary veins may be identified via ak-means clustering algorithm, where the branches of the tree arepartitioned into multiple clusters, and each cluster corresponds to adifferent lobar branch. Areas identified as pulmonary veins subtrees areconsidered “high cost” areas within a “cost image” which is used as aninput to a watershed algorithm to determine the planes which areequidistant to the edges of each of the subtrees, or, analogously, whichare “lowest cost” by the input cost image.

Another method to identify lung lobes is based on the application of awatershed algorithm to airways. It takes as input a segmentation of theairways. It then identifies each lobar branch of the airways (and thesubtree associated with such branch) using the following approach:airway branches that appear within the segmented airways are matched tocommon branching variants of lung anatomy contained within predefinedabstract representations of airway anatomy. Alternatively, lobarbranches of airways may be identified via a k-means clusteringalgorithm, where the branches of the tree are partitioned into multipleclusters, and each cluster corresponds to a different lobar branch.Areas identified as airway subtrees are considered “high cost” areaswithin a “cost image” which is used as an input to a watershed algorithmto determine the planes which are equidistant to the edges of each ofthe subtrees, or, analogously, which are “lowest cost” by the input costimage. An example of a technique to determine lobar branches of airwaysis described by Gu 2012.

Another method to identify lung lobes is based on the application of awatershed algorithm to a combination of at least two of pulmonaryarteries, pulmonary veins, and airways. Another method to identify lunglobes is based on any of the methods discussed above in combination witha cost image where areas with fissures are also deemed “high cost”.

Another way to identify the lung lobes is with atlas segmentation. Inthis case, the atlas contains source medical images (CT, MR, etc.)and/or 3D reconstructions of lungs. The 3D reconstructions may containthe lung surface and may also contain arteries, airways, and/or veins.The input to be compared to the atlas may be a standard medical image(CT, MR, etc.) and/or a 3D reconstruction of such image. The atlas imagethat is most similar to the input image is determined. Then thetransformation from the atlas image to the input image is determined.Then that transformation is applied, in reverse, to the loberepresentation of the atlas image to arrive at the lobe representationof the input lung image.

Another method to determine lung lobes is to combine any number of theabove methods. Examples of similar techniques are described by Lassen2013 and Giuliani 2018.

VIII. Identification of Segment Boundaries

This section describes the present disclosure, which is a system andmethod that enables a computer system to identify and label lungsegments from a three-dimensional medical image, and to visually depictsuch lung segments within a 3D reconstruction.

This paragraph and the next discuss identifying segment boundaries bylocating planes that represent the boundaries between adjacent lungsegments. One method begins by analyzing a representation of a lobe andits pulmonary artery subtree to identify the segmental branches of suchsubtree, as in the following approach: arterial branches that appearwithin the segmented lobar subtree of pulmonary arteries are matched tocommon branching variants of lung anatomy contained within predefinedabstract representations of arterial anatomy. Alternatively, segmentalbranches of pulmonary arteries may be identified via a k-meansclustering algorithm, where the branches of the lobar subtree arepartitioned into multiple clusters, and each cluster corresponds to adifferent segmental branch. Then, after identifying those segmentalbranches, it determines the lobe's segmental artery subtree. Then, itidentifies the planes that are equidistant between each of the arterialsubtrees that are contained in adjacent lung segments. Each identifiedplane represents at least a portion of the boundary between suchsegments.

Another method begins by analyzing a representation of a lobe and itsairway subtree to identify the segmental branches of such subtree as inthe following approach: airway branches that appear within the segmentedlobar subtree of airways are matched to common branching variants oflung anatomy contained within predefined abstract representations ofairway anatomy. Alternatively, segmental branches of airways may beidentified via a k-means clustering algorithm, where the branches of thelobar subtree are partitioned into multiple clusters, and each clustercorresponds to a different segmental branch. Then, after identifyingthose segmental branches, it determines the lobe's segmental airwaysubtree. Then, it identifies the planes that are equidistant betweeneach of the airway subtrees that are contained in adjacent lungsegments. Each identified plane represents at least a portion of theboundary between such segments.

This paragraph and a few subsequent paragraphs discuss identifyingsegment boundaries by identifying planes based on the location ofintersegmental veins. This method analyzes a representation of a lobeand a representation of the lobe's pulmonary vein subtree to identifythe intrasegmental and intersegmental veins within that lobe as in thefollowing approach: venous branches that appear within the segmentedlobar subtree of pulmonary veins are matched to common branchingvariants of lung anatomy contained within predefined abstractrepresentations of venous anatomy. Each identified intersegmental veinbranch is a line or other path which lies on the plane that defines theboundary between two adjacent segments (the “intersegmental plane”).After identifying this line, the system identifies such plane using,e.g., one of the methods described in the following paragraphs.

The first example method to identify the intersegmental plane is via awatershed algorithm, which assumes that the plane is equidistant to theedges of subtrees of intrasegmental veins, and/or subtrees of pulmonaryarteries, and/or subtrees of airways. The plane is then adjusted viasurface fitting such that the intersegmental vein will lie on theresulting plane. Examples of surface fitting techniques include leastsum of squares fitting, linear and non-linear regression, interpolation,RANSAC, B-Spline surface fitting as described in Liew 2015, and Houghtransformations as described by Drost 2015.

The second example method to identify intersegmental planes is via atlassegmentation and adjusted via surface fitting such that theintersegmental veins will lie on the resulting plane.

The third example method to identify intersegmental planes is via bymachine learning and adjusting via surface fitting such that theintersegmental veins will lie on the resulting plane.

Another method to generate segment boundaries is via machine learningtechniques, directly. In this approach, the machine learning model hasbeen trained with training inputs of three dimensional representationsof lungs, which might contain any of: lung surfaces, lung parenchyma,veins, arteries, airways, and fissures, and for which the output is lungsegments. This model takes in a three-dimensional representation of alung without segment boundaries and generates an output that includessegment boundaries.

Another method to generate segment boundaries is with atlassegmentation. Here, the input considered is a three-dimensionalrepresentation of a lobe and the atlas constitutes a set ofthree-dimensional representations of lobes with their correspondingsegment boundaries identified. The method compares the input lobethree-dimensional representation to the atlas of lobe three-dimensionalrepresentations with segment boundaries identified, determining the mostsimilar lobe three-dimensional representation within the atlas. Thetransformation from the atlas lobe to the input lobe is determined. Thesegment boundaries within the atlas lobe have that transformationapplied to generate the segment boundaries within the input lobe. Insome embodiments, the atlas constitutes a set of three-dimensionalrepresentations of a lobe and one or more of airways, arteries, andveins with their corresponding segment boundaries identified.

Another method to generate segment boundaries is by analyzing thedensity of voxels corresponding to vasculature and airways. Anycombination of following lobar subtrees may be considered: lobar subtreeof airways, lobar subtree of arteries, lobar subtree of veins. Thisanalysis identifies the planes on the interior of the lung lobe byidentifying the airways and/or pulmonary arteries and in particularthose areas that have the lowest level of voxel density; a low level ofvoxel intensity indicates a “channel” through which the plane will run.If considering intersegmental veins, an opposite approach is undertaken;a high level of voxel intensity is indicative of an intersegmental vein,which in turn is indicative of a segment boundary.

If desired, any number of the above-described techniques, such as voxeldensity analysis, atlas segmentation, machine learning, watershedbetween subtrees, intersegmental vein location, and surface fitting, maybe performed in combination.

IX. Flowcharts of Example Methods of Identifying Lung Features

FIGS. 5A and 5B present a flow chart of one embodiment of a method 500for determining segment boundaries of a lung. As shown, method 500initially obtains or receives image data for a lung (operation 502). Asindicated, this image data may take the form of voxels or otherrepresentation of points in a three-dimensional space. As indicatedherein, such data may be obtained from various sources such as x-rayimages (e.g., CT scans), nuclear magnetic resonance images, and othermedical imaging data such as those obtained by positron emissiontomography. In various embodiments, the data includes image informationfor at least an entire segment of a lung.

As illustrated in an operation 504, the method segments and labels theimage data provided in operation 502 to identify those voxels or otherpoints that correspond to the location of pulmonary arteries within alobe or lung. Also, as illustrated in an operation 506, the methodsegments and labels the voxel data obtained in operation 502 to identifythose voxels that correspond to locations of pulmonary veins. Further,in an operation 508, the method segments and labels the image dataobtained in operation 502 to identify voxels corresponding to locationsof pulmonary airways or airway tissue within a lobe or lung. Also, in anoperation 510, the method segments and labels image data obtained inoperation 502 to identify voxels within the three-dimensional space ofthe lobe or lung where parenchyma is located. Further, in an operation512, the method segments and labels the image data obtained in operation502 and identifies those voxels that correspond to locations of fissuresin the lung.

To this point, method 500 has analyzed three-dimensional image data of alobe or lung to identify various anatomical features. In the depictedembodiment, these features are pulmonary arteries, pulmonary veins,pulmonary airways, parenchyma, and lobe or lung fissures. This is not anexhaustive list. The method may analyze the image data to identify oneor more other anatomical features such as lymph nodes or lesions.Further, in some implementations, the method does not identify each ofthe pulmonary arteries, pulmonary veins, pulmonary airways, parenchyma,and lobe or lung fissures. In other words, while the depicted embodimentshows that operations 504 through 512 are performed in the depictedmethod, this is not required. In some embodiments, any one or more ofthese anatomical features are segmented and labeled for use insubsequent operations. In certain embodiments, the fissures, theairways, the parenchyma, and the arteries are segmented and labeled,while the veins are not segmented and labeled. In some embodiments, theveins, parenchyma, and airways are segmented and labeled, while thearteries and fissures are not segmented and labeled. In someembodiments, the parenchyma, the fissures, the veins, and the arteriesare segmented and labeled, while the airways are not segmented andlabeled. In some embodiments, the parenchyma, the fissures, the veins,and the airways are segmented and labeled, while the arteries are notsegmented and labeled. In some embodiments, the parenchyma, the airways,the veins, and the arteries are segmented and labeled, while thefissures are not segmented and labeled. In some embodiments, theparenchyma, the arteries, and the airways are segmented and labeled,while the veins and fissures are not segmented and labeled.

In the depicted embodiment, the method next uses one or more of thesegmented and labeled anatomical features including the lung fissures,the parenchyma, the pulmonary airways, the pulmonary veins, and/or thepulmonary arteries in order to locate lobe boundaries in thethree-dimensional space representing the lung (operation 514). Incertain embodiments, the method employs only a subset of the listedoperations to identify lobe boundaries. For example, the method mayemploy only the lung fissures, parenchyma, and arteries. In certainembodiments, and depending on the lung under consideration, operation514 may be repeated to locate multiple lobe boundaries in the lung underconsideration.

Next, in the depicted embodiment, at an operation 516, the methodidentifies segmental branches and related subtrees of the pulmonaryarteries identified in operation 504. Additionally, in the depictedembodiment, at an operation 518, the method identifies segmentalbranches and related subtrees of the pulmonary arteries identified inoperation 508. When the method identifies the branch that leads into agiven segment, it can trace the artery network downstream to identifythe full tree of an arterial or airway network within a segment.Further, in an operation 520 of the depicted process, the methodidentifies one or more intersegmental branches of veins within lobes.The intersegmental branches are identified from the veins identified inoperation 506.

Next, in an operation 522 of the depicted embodiment, the method usesany one or more of the segmental features identified in operations 516,518, and 520, optionally along with the parenchyma, to determine atleast a portion of the boundaries for a given segment. Note that theseoperations 516, 518, 520, and 522 are performed within a given lobe,which contains multiple segments. Operation 522 identifies a boundarybetween two adjacent segments in a lobe. It does not necessarilyidentify the full boundary of a segment. In various embodiments anddepending on how many segments reside in a given lobe, the methodrepeats operation 522 to generate intersegmental boundaries betweenmultiple segments (sometimes all segments) in a lobe underconsideration.

Next, in an operation 524 of the depicted embodiment, the method usessome or all of the intersegmental boundaries identified in operation 522along with some or all of the lobe boundaries identified in operation514 to identify the all or substantially all of the lung segmentboundaries for at least one segment within a lobe.

Finally, in an operation 526, the method generates a representation ofthe lung or lobe showing some or all the segment boundaries identifiedin operation 524, optionally along with lobe boundaries identified inoperation 514.

Note that the segment boundaries include interior portions within a lobeinterior and surface portions on the lobe surface. In some cases, themethod uses the same technique to identify both the interior and surfaceportions of the segment boundaries. The relevant technique is employedin operation 522 of the depicted embodiment.

FIGS. 6A and 6B present a flow chart of another embodiment of a method500 for identifying and optionally representing boundaries of one ormore lung segments. Some of the operations are similar to those shown inFIGS. 5A and 5B. For example, operation 510 (segmenting and labelingparenchyma) may be implemented using Otsu's thresholding. As anotherexample, the operations 504 and 506 that involve segmenting and labelingvoxels corresponding to arteries and veins may be implemented byperforming tubularity analysis to identify voxels likely to bevasculature, followed by assembling such voxels (tubular collections ofvoxels) into subtrees using a non-linear program, and, ultimately,labelling the subtrees as veins and arteries using a non-linear program.As a further example, operation 508, which involves segmenting andlabelling voxels corresponding to airways may be implemented using amachine learning model with all image voxels as inputs.

In method 600, lobe boundaries are identified by fitting planes to (1)areas of low airway, vein, and/or artery voxel density, and (2) highfissure voxel density. The method also identifies branches of arteriesand arterial subtrees within particular segments of a lobe. The methodalso identifies branches of airways and airway subtrees withinparticular segments of a lobe. Still further, the method also identifiesintersegmental veins within particular segments of a lobe. In thedepicted embodiment, the method uses cost images and a watershedalgorithm to identify planes between segmental airways and arterialsub-trees. The voxels of airways and arteries are rendered as high costvoxels.

In the depicted embodiment, the method then fits the planes tointersegmental veins between adjacent segments. This allows the methodto identify boundaries between adjacent segments within the same lobe.The method then uses these boundaries and the lobe boundaries to definefull lung segment boundaries. Ultimately, the method may generate arepresentation of the lung or lobe with substantially complete segmentboundaries.

As shown, method 600 may include one or more of, as examples, obtaininglung image data (operation 602); determining voxels corresponding toparenchyma, optionally via Otsu's thresholding (operation 604);performing tubularity analysis to determine voxels likely to bevasculature (operation 606); assembling tubular voxels into vasculaturesubtrees by solving a non-linear program (operation 608); labelingvasculature subtrees as veins and arteries by solving a non-linearprogram (610); determining airway voxels and subtrees via a machinelearning model with all image voxels as input (operation 612);identifying lobe boundaries by fitting planes to the areas of lowestairway, vein, and/or artery voxel density and/or the highest fissurevoxel density (operation 614); identifying segmental branches ofarteries and related vascular subtrees within lobes (operation 616);identifying segmental branches of airways and related subtrees ofairways within lobes (operation 618); identifying intersegmental veinswithin lobes (operation 620); initialize a cost image for each segmentalboundary between two adjacent segments within a lobe (operation 622);updating each cost image, rendering voxels of arterial and airwaysubtrees as high cost (operation 624); running a watershed algorithm toidentify planes equidistance to segmental airway and artery subtrees(operation 626); fitting planes to intersegmental veins between adjacentsegments to identify boundaries between segments within the same lobe(operation 628); using boundaries between segments within the same lobeand lobe boundaries to define full lung segment boundaries (operation630); and generating a representation of the lung with identification ofthe lung lobes and substantially complete segment boundaries (operation632).

Note that method 600 need not be implemented as an example of the method500. The operations presented in method 600 need not correspond toparticular operations in method 500. The above discussion linking someoperations of method 600 to operations of method 500 is provided as anexample.

The methods of FIGS. 5A, 5B, 6A, and 6B may be implemented using variousforms of image processing or image analysis logic such any combinationof the computational techniques described above. The image analysislogic may be hosted on a single computational machine or distributedacross multiple machines, optionally including some components thatexecute locally and others that execute remoted, e.g., via cloud-basedresources.

X. Additional Details

The various processes, algorithms, software, etc. disclosed herein maybe implemented with, executed on, or otherwise performed by a computingsystem having various hardware and software elements.

The hardware components may include processors, controllers, storagedevices, and the like and may be implemented locally or remotely(including on the cloud).

Software components may be expressed (or represented) as data and/orinstructions embodied in various computer-readable media, in terms oftheir behavioral, logical, and/or other characteristics.Computer-readable media in which such formatted data and/or instructionsmay be embodied include, but are not limited to, non-volatile storagemedia in various forms (e.g., optical, magnetic or semiconductor storagemedia). The computer-readable media corresponding to such systems arealso intended to fall within the scope of the present disclosure.

Reference herein to “one embodiment” or “an embodiment” means that aparticular feature, structure, or characteristic described in connectionwith the embodiment can be included in one some or all of theembodiments of the present disclosure. The usages or appearances of thephrase “in one embodiment” or “in another embodiment” in thespecification are not referring to the same embodiment, nor are separateor alternative embodiments necessarily mutually exclusive of one or moreother embodiments. The same applies to the term “implementation.” Thepresent disclosure is neither limited to any single aspect norembodiment thereof, nor to any combinations and/or permutations of suchaspects and/or embodiments. Moreover, each of the aspects of the presentdisclosure, and/or embodiments thereof, may be employed alone or incombination with one or more of the other aspects of the presentdisclosure and/or embodiments thereof. For the sake of brevity, certainpermutations and combinations are not discussed and/or illustratedseparately herein.

Further, an embodiment or implementation described herein as exemplaryis not to be construed as preferred or advantageous, for example, overother embodiments or implementations; rather, it is intended to conveyor indicate that the embodiment or the embodiments are exampleembodiment(s).

In the examples, the term “determine” and other forms thereof (i.e.,determining, determined and the like or calculating, calculated and thelike) means, among other things, calculate, assesses, determine and/orestimate and other forms thereof.

In addition, the terms “first,” “second,” and the like, herein do notdenote any order, quantity, or importance, but rather are used todistinguish one element from another. Moreover, the terms “a” and “an”herein do not denote a limitation of quantity, but rather denote thepresence of at least one of the referenced item. Further, the terms“data” and “metadata” may mean, among other things information, whetherin analog or a digital form (which may be a single bit (or the like) ormultiple bits (or the like)).

As used in the examples, the terms “comprises,” “comprising,”“includes,” “including,” “have,” and “having” or any other variationthereof, are intended to cover a non-exclusive inclusion, such that aprocess, method, article, or apparatus that comprises a list of elementsdoes not include only those elements but may include other elements notexpressly listed or inherent to such process, method, article, orapparatus.

The example elements that do not recite “means” or “step” are not in“means plus function” or “step plus function” form. (See, 35 USC §112(f)). Applicant's intend that only example elements reciting “means”or “step” be interpreted under or in accordance with 35 U.S.C. § 112(f).

In the foregoing description, numerous specific details are set forth toprovide a thorough understanding of the presented implementations. Thedisclosed implementations may be practiced without some or all of thesespecific details. In other instances, well known process operations havenot been described in detail to not unnecessarily obscure the disclosedimplementations. While the disclosed implementations are described inconjunction with the specific implementations, it will be understoodthat it is not intended to limit the disclosed implementations.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, it will be apparent thatcertain changes and modifications may be practiced within the scope ofthe appended examples. It should be noted that there are manyalternative ways of implementing the processes, systems, and apparatusof the present embodiments. Accordingly, the present embodiments are tobe considered as illustrative and not restrictive, and the embodimentsare not to be limited to the details given herein.

XI. Additional Disclosures

The following publications are hereby incorporated herein by referencein their entireties. To the extent that these references are present inthe body of this disclosure, they are incorporated for at least thepurpose and/or context presented in corresponding disclosure.

Benmansour, Fethallah, and Laurent D. Cohen. “Tubular structuresegmentation based on minimal path method and anisotropic enhancement.”International Journal of Computer Vision 92.2 (2011): 192-210.

Benmansour, Fethallah, Engin Türetken, and Pascal Fua. Tubular geodesicsusing oriented flux: An ITK implementation. No. ARTICLE. 2013.

Gu, Suicheng, et al. “Automated lobe-based airway labeling.” Journal ofBiomedical Imaging 2012 (2012): 1.

Graham, Michael W., et al. “Robust 3-D airway tree segmentation forimage-guided peripheral bronchoscopy.” IEEE transactions on medicalimaging 29.4 (2010): 982-997.

Helmberger, Michael, et al. “Quantification of tortuosity and fractaldimension of the lung vessels in pulmonary hypertension patients.” PloSone 9.1 (2014): e87515.

Aurenhammer, Franz. “Voronoi diagrams—a survey of a fundamentalgeometric data structure.” ACM Computing Surveys (CSUR) 23.3 (1991):345-405.

Cornea, Nicu D., et al. “Computing hierarchical curve-skeletons of 3Dobjects.” The Visual Computer 21.11 (2005): 945-955.

van Ginneken, Bram, Wouter Baggerman, and Eva M. van Rikxoort. “Robustsegmentation and anatomical labeling of the airway tree from thoracic CTscans.” International Conference on Medical Image Computing andComputer-Assisted Intervention. Springer, Berlin, Heidelberg, 2008.

Meng, Qier, et al. “Airway segmentation from 3D chest CT volumes basedon volume of interest using gradient vector flow.” Medical ImagingTechnology 36.3 (2018): 133-146.

Antiga, Luca. “Generalizing vesselness with respect to dimensionalityand shape.” The Insight Journal 3 (2007): 1-14.

Lassen, Bianca, et al. “Automatic segmentation of the pulmonary lobesfrom chest CT scans based on fissures, vessels, and bronchi.” IEEEtransactions on medical imaging 32.2 (2012): 210-222.

Law, Max WK, and Albert CS Chung. “Three dimensional curvilinearstructure detection using optimally oriented flux.” European conferenceon computer vision. Springer, Berlin, Heidelberg, 2008.

Giuliani, Nicola, et al. “Pulmonary Lobe Segmentation in CT Images usingAlpha-Expansion.” VISIGRAPP (4: VISAPP). 2018.

Payer, Christian, et al. “Automatic artery-vein separation from thoracicCT images using integer programming.” International Conference onMedical Image Computing and Computer-Assisted Intervention. Springer,Cham, 2015.

Payer, Christian, et al. “Automated integer programming based separationof arteries and veins from thoracic CT images.” Medical image analysis34 (2016): 109-122.

Pu, Jiantao, et al. “A differential geometric approach to automatedsegmentation of human airway tree.” IEEE transactions on medical imaging30.2 (2010): 266-278.

Shekhovtsov, Alexander, Paul Swoboda, and Bogdan Savchynskyy. “Maximumpersistency via iterative relaxed inference with graphical models.”Proceedings of the IEEE

Conference on Computer Vision and Pattern Recognition. 2015.

Turetken, Engin, et al. “Reconstructing loopy curvilinear structuresusing integer programming.” Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition. 2013.

Türetken, Engin, et al. “Reconstructing curvilinear networks using pathclassifiers and integer programming.” IEEE transactions on patternanalysis and machine intelligence 38.12 (2016): 2515-2530.

van Dongen, Evelien, and Bram van Ginneken. “Automatic segmentation ofpulmonary vasculature in thoracic CT scans with local thresholding andairway wall removal.” 2010 IEEE International Symposium on BiomedicalImaging: From Nano to Macro. IEEE, 2010.

Ochs, Robert A., et al. “Automated classification of lungbronchovascular anatomy in CT using AdaBoost.” Medical image analysis11.3 (2007): 315-324.

Rohlfing, Torsten, et al. “Evaluation of atlas selection strategies foratlas-based image segmentation with application to confocal microscopyimages of bee brains.” NeuroImage 21.4 (2004): 1428-1442.

Beucher, Serge. “The watershed transformation applied to imagesegmentation.” SCANNING MICROSCOPY-SUPPLEMENT-(1992): 299-299.

Helen, R., et al. “Segmentation of pulmonary parenchyma in CT lungimages based on 2D Otsu optimized by PSO.” 2011 International Conferenceon Emerging Trends in Electrical and Computer Technology. IEEE, 2011.

Shirk, Joseph D., et al. “Effect of 3-Dimensional Virtual Reality Modelsfor Surgical Planning of Robotic-Assisted Partial Nephrectomy onSurgical Outcomes: A Randomized Clinical Trial.” JAMA network open 2.9(2019): e1911598-e1911598.

Drost, Bertram, and Slobodan Ilic. “Local hough transform for 3dprimitive detection.” 2015 International Conference on 3D Vision. IEEE,2015.

XII. CONCLUSION

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, it will be apparent thatcertain changes and modifications may be practiced within the scope ofthe appended claims. It should be noted that there are many alternativeways of implementing the processes, systems, and apparatus of thepresent embodiments. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and the embodiments arenot to be limited to the details given herein.

What is claimed is:
 1. A method of determining segment boundaries oflung segments, the method comprising: receiving image data forming athree-dimensional representation of at least a part of a lung;computationally identifying, using the image data, at least oneanatomical feature within the lung, wherein the at least one anatomicalfeature comprises one or more of the following: a network of arteries, anetwork of veins, a network of airways, some or all of lung parenchyma,and one or more fissures of the lung; computationally identifying, usingthe at least one anatomical feature, a segment boundary of at least onelung segment within the lung, wherein each lobe in the lung is comprisedof one or more lung segments; and generating a representation of all orpart of the lung showing the segment boundary of the at least one lungsegment within the lung.
 2. The method of claim 1, wherein the imagedata comprises a one or both of a CT scan and an Mill.
 3. The method ofclaim 1, wherein the at least one anatomical feature comprises some orall of the lung parenchyma and the network of arteries.
 4. The method ofclaim 1, wherein the at least one anatomical feature comprises some orall of the lung parenchyma and the network of veins.
 5. The method ofclaim 1, wherein the at least one anatomical feature comprises some orall of the lung parenchyma and the network of airways.
 6. The method ofclaim 1, wherein the at least one anatomical feature comprises the oneor more fissures of the lung.
 7. The method of claim 1, whereincomputationally identifying the segment boundary of the at least onelung segment comprises: computationally identifying, using the at leastone anatomical feature, one or more of the following: segmental branchesof arteries and related subtrees, segmental branches of veins, segmentalbranches of airways and related subtrees, and the fissures of the lung;generating segment boundaries between adjacent lung segments in the lungthrough analysis of segmental branches of arteries and related subtrees,segmental branches of veins, segmental branches of airways and relatedsubtrees, and/or fissures of the lung; and combining the segmentboundaries between adjacent lung segments to define the boundary of theat least one lung segment within the lung.
 8. The method of claim 1,further comprising: computationally identifying, using the at least oneanatomical feature, lobe boundaries for a plurality of lobes in thelung.
 9. The method of claim 1, further comprising: computationallyidentifying, using the image data, a lesion within the lung; andcomputationally determining, from the image data, that the lesion islocated in a given lung segment of the at least one lung segment or in aregion spanning two or more lung segments.
 10. The method of claim 9,further comprising: computationally measuring, from the image data, aminimum distance between the lesion and the segment boundary of any ofthe one or more lung segments.
 11. A method of identifying segmentboundaries of lung segments, the method comprising: receiving image dataforming a volumetric representation of at least a part of a lung;computationally identifying, using the image data, a network ofarteries; computationally identifying, using the image data, a networkof bronchi; computationally identifying, using the image data, a networkof veins; and computationally identifying, based on the identifiednetwork of arteries, network of bronchi, and network of veins, segmentboundaries of a plurality of lung segments within at least one lobe ofthe lung, wherein each lobe in the lung is comprised of one or more lungsegments.
 12. The method of claim 11, wherein the image data comprisesone or both of a CT scan and an Mill.
 13. The method of claim 11,wherein computationally identifying the network of arteries comprises:computationally identifying a tube-like structure in the image data,wherein the tube-like structure is identified by identifying a set ofgradient changes within the image data; and computationally determining,using the image data and based on how the tube-like structure brancheswithin the lung, that the tube-like structure is part of the network ofarteries.
 14. The method of claim 11, wherein computationallyidentifying the network of bronchi comprises: computationallyidentifying a tube-like structure in the image data, wherein theadditional tube-like structure is identified by identifying a set ofgradient changes within the image data; and computationally determining,using the image data and based on how the additional tube-like structurebranches within the lung, that the tube-like structure is part of thenetwork of bronchi.
 15. The method of claim 11, wherein computationallyidentifying the network of veins comprises: computationally identifyinga tube-like structure in the image data, wherein the tube-like structureis identified by identifying a set of gradient changes within the imagedata; and computationally determining, using the image data and based onhow the tube-like structure branches within the lung, that the tube-likestructure is part of the network of veins.
 16. The method of claim 11,wherein computationally identifying the segment boundary of a singlelung segment in the plurality of lung segments comprises:computationally identifying a volume within the image data that receivesblood from a section of the network of arteries.
 17. The method of claim11, wherein computationally identifying a segment boundary of a singlelung segment in the plurality of lung segments comprises:computationally identifying a volume within the image data that receivesair from a section of the network of bronchi.
 18. The method of claim11, further comprising: computationally identifying, using the imagedata, one or more intersegmental veins; and computationally refining,based on the one or more intersegmental veins, the segment boundaries ofthe plurality of lung segments of the at least one lobe.
 19. A method ofdetermining segment boundaries of lung segments, the method comprising:receiving image data forming a three-dimensional representation of atleast a part of a lung; computationally identifying, using the imagedata, one or more intersegmental veins within the lung; computationallyidentifying at least part of a segment boundary of at least one lungsegment within the lung based at least in part on the identified one ormore intersegmental veins, wherein each lobe in the lung is comprised ofone or more lung segments; and generating a representation of all orpart of the lung showing at least part of the segment boundary of the atleast one lung segment within the lung.
 20. A method of determiningsegment boundaries of lung segments, the method comprising: receivingimage data forming a three-dimensional representation of at least a partof a lung; computationally identifying, using the image data, one ormore fissures of the lung; computationally identifying at least part ofa segment boundary of at least one lung segment within the lung based atleast in part on the identified one or more fissures, wherein each lobein the lung is comprised of one or more lung segments; and generating arepresentation of all or part of the lung showing at least part of thesegment boundary of the at least one lung segment within the lung.